{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 3 - Plotting *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Theory of Visualisation and EDA\n",
    "2. Basic Plotting with Python\n",
    "3. Graphs types\n",
    "4. Summary statistics and box plots\n",
    "\n",
    "\\* [1] Data Analysis with Open Source Tools by Philipp K. Janert. Nearly all of the text, examples and datasets are directly sourced from the chapter 2 from the book.\n",
    "\n",
    "\\* [2] Some examples taken from http://pandas.pydata.org/pandas-docs/stable/visualization.html#plotting-on-a-secondary-y-axis\n",
    "\n",
    "http://nb.bianp.net/sort/views/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://shop.oreilly.com/product/9780596802363/ViewLarger.do?sortby=bestSellers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Outcomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of this lecture, you should be able to:\n",
    "\n",
    "* describe the role visualisation and of EDA\n",
    "* describe and list the primary plotting libraries available through Python\n",
    "* plot univariate and multivariate data using a variety of graphs: histograms, KDEs, CDFs, scatter, box, line\n",
    "* explain the pitfalls of relying on simply summary statistics\n",
    "* apply appropriate graphs to given data types\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual representation of data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Data visualization is the graphical display of abstract information for two purposes: sense-making (also called data analysis) and communication.**\n",
    "\n",
    "> Data visualization is only successful to the degree that it encodes information in a manner that our eyes can discern and our brains can understand. Getting this right is much more a science than an art, which we can only achieve by studying human perception. The goal is to translate abstract information into visual representations that can be easily, efficiently, accurately, and meaningfully decoded. \n",
    "\n",
    "To achieve this goal, the display should achieve the following:\n",
    "\n",
    "*    Clearly indicates how the values relate to one another\n",
    "*    Represents the quantities accurately\n",
    "*    Makes it easy to compare the quantities\n",
    "*    Makes it easy to see the ranked order of values, such as from the leading cause of death to the least\n",
    "*    Makes obvious how people should use the information - what they should use it to accomplish - and encourages them to do this\n",
    "    \n",
    "Source: https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/data-visualization-for-human-perception\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "    \n",
    "    \n",
    "> “Graphical excellence is that which gives to the viewer the greatest number of ideas in the shortest time with the least ink in the smallest space.”\n",
    "― Edward R. Tufte, The Visual Display of Quantitative Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How not to do graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "“Since the turn of the 21st century, data visualization has been popularized, too often in tragically ineffective ways as it has reached the masses through commercial software products.\"\"\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "<img src=https://www.biostat.wisc.edu/~kbroman/topten_worstgraphs/cotter_fig2.jpg width=800>\n",
    "\n",
    "\n",
    "<img src=https://www.interaction-design.org/images/encyclopedia/data_visualization_for_human_perception/three-dimensional-bar-chart-user-interface.jpg width=800>\n",
    "\n",
    "<img src=https://public-media.interaction-design.org/images/encyclopedia/data_visualization_for_human_perception/pie-chart-distorted-causes-of-death.jpg width=800>\n",
    "\n",
    "\n",
    "<img src=https://www.biostat.wisc.edu/~kbroman/topten_worstgraphs/roeder_fig4.jpg width=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How we perceive visual cues "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Available visual encoding schemes:\n",
    "\n",
    "<img src=../figures/imageEncodingSchemes.png width=800>\n",
    "\n",
    "Accuracy of various visual encoding schemes for conveying quantitative data:\n",
    "\n",
    "<img src=../figures/accuracyOfVisualPerception.png width=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which data types map best to which visual encoding schemes:\n",
    "\n",
    "<img src=../figures/mappingDataTypesToEncoding.png width=800>\n",
    "\n",
    "Source: http://mkweb.bcgsc.ca/vizbi/2012/principles.pdf\n",
    "\n",
    "Mackinlay, J. (1986). Automating the design of graphical presentations of relational information. Acm Transactions On Graphics (Tog), 5(2), 110-141."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some principles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> \"Clutter and confusion are not attributes of data - they are shortcomings of design.\" ― Edward R. Tufte"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<video controls alt=\"test\" src='../figures/ImprobableFemaleBasenji.webm'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "> “There is no such thing as information overload. There is only bad design.“\n",
    "― Edward R. Tufte\n",
    "\n",
    "\n",
    "> “Above all else show the data.”\n",
    "― Edward R. Tufte, The Visual Display of Quantitative Information\n",
    "\n",
    "\n",
    "> “If the statistics are boring, then you've got the wrong numbers.”\n",
    "― Edward R. Tufte\n",
    "\n",
    "\n",
    "> “Clutter and confusion are failures of design, not attributes of information.”\n",
    "― Edward R. Tufte\n",
    "\n",
    "\n",
    "> “It is not how much empty space there is, but rather how it is used. It is not how much information there is, but rather how effectively it is arranged.”\n",
    "― Edward R. Tufte \n",
    "\n",
    "\n",
    "> “Colour  used poorly is worse than no  colour  at all”\n",
    "― Edward Tufte\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When examining data clouds, some of the **key characteristics** that need to be considered are:\n",
    "\n",
    "1. the **location** of the data and its centre of mass\n",
    "2. the **dispersion** or spread of the data\n",
    "3. the **shape** of the data\n",
    "\n",
    "The two graphs below gives a good indication of the above:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "matplotlib is the foundational and the most popular Python library for producing plots and other 2D data visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"http://matplotlib.org/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1a130859a58>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML, IFrame\n",
    "IFrame(\"http://matplotlib.org/\", width=800, height=600)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The library is imported into your code as follows:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import matplotlib.pyplot as plt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "matplotlib is very powerful since all aspects of a figure can be controlled programmatically and is therefore highly suitable for generating figures particularly in scientific publications but also in business settings. This is important for reproducibility and convenient when one needs to regenerate a figure with updated data or modify its appearance. \n",
    "\n",
    "Despite its expressive power, matplotlib also has it's disadvantages. Firstly, because the library enables the user to tweak all aspects of the figures, the user is at first overwhelmed with the number of tunable knobs. Secondly, the default matplotlib figures are ugly. It takes a lot of effort (and knowledge) to make the figures aesthetically pleasing.\n",
    "\n",
    "There are however a number of plotting libraries out there that use matplotlib under the hood, but provide the user with an easier interface to it, and importantly produce default figures which are visually much more appealing. Seaborn is one such library that we will be using."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import seaborn as sns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"https://web.stanford.edu/~mwaskom/software/seaborn/index.html\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1a130859d30>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML, IFrame\n",
    "IFrame(\"https://web.stanford.edu/~mwaskom/software/seaborn/index.html\", width=1000, height=600)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because seaborn uses matplotlib, the graphics can be further tweaked using matplotlib interfaces and rendered with any of the matplotlib backends to generate publication-quality figures.Examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=../figures/bubble.png width=800>\n",
    "<img src=../figures/seabornGraph.png width=800>\n",
    "<img src=\"../figures/Usage of technologies amongst participants2.png\" width=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternate library is pylab."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import pylab\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pylab provides an interface that is very similar to MATLAB, therefore experienced users of MATLAB can make an easy transition to python plotting. We will only minimally make use of Pylab in order to set global appearance settings for the plotting environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, there is an interactive plotting environment called Bokeh."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"https://bokeh.pydata.org/en/latest/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1a130859e10>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"https://bokeh.pydata.org/en/latest/\", width=1000, height=600)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting Univariate and Multivariate Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set(style=\"ticks\")\n",
    "from pylab import rcParams\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#don't forget this line\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matplotlib.style.use('ggplot')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a set of tunable parameters for controlling the aesthetics of the graphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rcParams['figure.figsize'] = 15, 10\n",
    "rcParams['font.size'] = 20\n",
    "rcParams['axes.facecolor'] = 'white'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rcParams['figure.dpi'] = 350\n",
    "#rcParams['lines.linewidth'] = 2\n",
    "#rcParams['patch.edgecolor'] = 'white'\n",
    "#rcParams['font.family'] = 'StixGeneral'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 1 - Server Response Times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin with the example from [1], using a dataset called \"ch02_serverdata\" containing server response times to requests in milliseconds."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "452.42\n",
    "318.58\n",
    "144.82\n",
    "129.13\n",
    "1216.45\n",
    "991.56\n",
    "1476.69\n",
    "662.73\n",
    "1302.85\n",
    "1278.55\n",
    "627.65\n",
    "1030.78\n",
    "215.23\n",
    "44.50\n",
    "133.410\n",
    "619.96\n",
    "970.97\n",
    "1062.71\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data into a data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data = pd.read_csv(\"../datasets/ch02_serverdata\",header=None)\n",
    "server_data.head(10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get some info on the data type and the number of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data.info()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column had no header so let's see what the default column name is in the data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data.columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 2 - Wine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#wine_data = pd.read_csv(\"wine_data.csv\")\n",
    "wine_data = pd.io.parsers.read_csv(\n",
    "    '../datasets/wine_data.csv',\n",
    "     usecols=[0,1,2,3,4,5]\n",
    "    )\n",
    "wine_data.columns=['Class','Alcohol','Malic acid','Ash','Alcalinity of ash','Magnesium']\n",
    "wine_data.head(10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_data.info()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_data.describe()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 3 - Tips"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the example tips dataset\n",
    "tips = sns.load_dataset(\"tips\")\n",
    "tips['percentage'] = tips.tip / tips.total_bill * 100.0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tips.info())\n",
    "tips.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to get a feel for the data and this can be best achieved visually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Plotting "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting methods allow for a handful of plot styles other than the default Line plot. These methods can be provided as the kind keyword argument to plot(). These include:\n",
    "\n",
    "    ‘bar’ or ‘barh’ for bar plots\n",
    "    ‘hist’ for histogram\n",
    "    ‘box’ for boxplot\n",
    "    ‘kde’ or 'density' for density plots\n",
    "    ‘area’ for area plots\n",
    "    ‘scatter’ for scatter plots\n",
    "    ‘hexbin’ for hexagonal bin plots\n",
    "    ‘pie’ for pie plots\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling plot on both dataframes will give us line graphs which use the index as the x-axis and each of the columns as y-axis values. This type of plot assumes a temporal relationship between each of the observations, which in our case is meaningless. We therefore need something more informative. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data.plot()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_data.plot()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Histograms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequenly it is useful to look at the distribution of data before you do anything to it. Histograms are a kind of bar graph that displays relative frequencies of data values; hence, the y-axis is always some measure of frequency. This can either be raw counts of values or scaled proportions. Histograms tend to be very good beginning points for EDA.\n",
    "\n",
    "However, they are weak on providing intuition, and crucially, they do not easily convey quantitative information from the graph. This is particularly amplified on larger datasets.\n",
    "\n",
    "A more informative graph type is needed.\n",
    "\n",
    "Histograms divide the range of values into a set of “bins” and then count the number of points that fall into each bin.\n",
    "\n",
    "The count for each bin is then plotted as a function of the position of the bin on the x-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data.hist()\n",
    "plt.title(\"Server Response Times\", alpha=0.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above histogram is rendered using default settings. The quality of any histogram hinges on the proper choice of bin width. Making width too large results in losing too much detailed information about the data set. If too small, then the shape of the distribution will not be discernible.\n",
    "\n",
    "Unfortunately, there is no simple rule of thumb exists to predict a perfect bin width for a given data set. Usually several different values for the bin width must be experimented with until a satisfactory result is obtained. Sometimes a rough guideline for determining the bins size is the square root on the dataset size.\n",
    "\n",
    "Below shows a histogram of this data set with the horizontal axis divided into 60 bins of 50 milliseconds width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data.hist(bins=60,facecolor='grey')\n",
    "plt.title(\"Server Response Times\", alpha=0.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Experiment with several different values for the bins parameter. \n",
    "\n",
    "Try using the square root of the dataset size as a parameter. \n",
    "\n",
    "Which do you think is more informative? Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The histogram tells us that:\n",
    "\n",
    "1. There is a rather sharp cut-off at a non zero value on the left, which means that there is a minimum completion time below which no request can be completed. \n",
    "2. Then there is a sharp rise to a maximum at the “typical” response time, and finally there is a relatively large tail on the right, corresponding to the smaller number of requests that take a long time to process.  \n",
    "\n",
    "Key point: it is important to realize that **a data set does not determine a histogram uniquely**. Instead, it is largely the bin width (and to some degree the alignment of the bins) which influence the shape of a histogram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can rescale any of the axes as required. Careful when performing this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "server_data.hist(bins=100,facecolor='grey')\n",
    "plt.title(\"Server Response Times\")\n",
    "plt.xscale('log')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(np.log(server_data[0]),facecolor='grey')\n",
    "plt.title(\"Server Response Times\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While histograms are common and possess an intuitive interpretation, they also have some serious limitations [1]:\n",
    "\n",
    "1. The binning process required by all histograms **loses information** (by replacing the location of individual data points with a bin of finite width). If we only have a few data points, we can ill afford to lose any information.\n",
    "\n",
    "2. Histograms are **not unique**. The appearance of a histogram can be quite different depending on the settings. (This non uniqueness is a direct consequence of the information loss described in the previous item.)\n",
    "\n",
    "3. On a more superficial level, histograms are **ragged and not **. This matters little if we just want to draw a picture of them, but if we want to feed them back into a computer as input for further calculations, then a smooth curve would be easier to handle.\n",
    "\n",
    "4. Histograms **do not handle outliers gracefully**. A single outlier, far removed from the majority of the points, requires many empty cells in between or forces us to use bins that are too wide for the majority of points. It is the possibility of outliers that makes it difficult to find an acceptable bin width in an automated fashion.\n",
    "\n",
    "There is an alternative to classical histograms that has none of these problems which we will look at later.\n",
    "\n",
    "In the multivariate case, we can get a sense of the distribution of the values for all of the variables at once (if there are not too many)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = wine_data.hist(bins=int(np.sqrt(len(wine_data))),facecolor='grey', figsize=(15, 15))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that we are dealing with a classification dataset, it makes sense to examine the distributions for various features by class label as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = wine_data['Alcohol'].hist(by=wine_data['Class'], figsize=(15, 15), sharey=True, sharex=True, bins=int(np.sqrt(len(wine_data))), color='grey')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Select the 'Malic acid' feature and generate a histogram for each of the class labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The FacetGrid class is useful when you want to visualize the distribution of a variable or the relationship between multiple variables separately within subsets of your dataset. \n",
    "\n",
    "The main approach for visualizing data on this grid is with the FacetGrid.map() method. Provide it with a plotting function and the name(s) of variable(s) in the dataframe to plot. \n",
    "\n",
    "Let’s look at the distribution of tip percentages according to smoker status using a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = sns.FacetGrid(tips, col=\"smoker\", height=6)\n",
    "g.map(plt.hist, \"percentage\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Plot a FacetGrid figure depicting the distribution of tip percentages according to sex using a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use multiple features to generate separate plots: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = sns.FacetGrid(tips, row='smoker', col=\"sex\", height=6)\n",
    "g.map(plt.hist, \"percentage\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kernel Density Estimates (KDEs) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "KDEs are a relatively new technique as they heavily rely on computing power to generate them.\n",
    "\n",
    "To form a KDE, a kernel (a smooth, strongly peaked function) is placed at the\n",
    "position of each data point. Then all the contributions from all kernels are added up to obtain a smooth curve which can be evaluated at any point along the x-axis.\n",
    "\n",
    "In KDEs, both the **type of a kernel and the bandwidth are tunable parameter**s. Given data set and bandwidth, a **KDE is unique**; a KDE is also smooth, provided we have chosen a smooth kernel function, such as the Gaussian.\n",
    "\n",
    "A KDE can be constructed directly from a data frame but it only allows for limited tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data[0].plot(kind='kde', title='Server Response Times', label='KDE pandas')\n",
    "plt.xlabel('Response Times')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The _bw_method_ parameter can be used to tune the bandwidth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bandwidth = 0.1\n",
    "server_data[0].plot(kind='kde', title='Server Response Times', label='KDE pandas',bw_method = bandwidth)\n",
    "plt.xlabel('Response Times')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seaborn has a tunable bandwidth for KDEs based on the Gaussian kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.kdeplot(server_data[0], bw=20, color='blue', lw=1.0, label='KDE seaborn')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plots can be superimposed into a single figure if executed in the same block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data.hist(bins=60, density=True, alpha=0.4, color='grey')\n",
    "server_data[0].plot(kind='kde', title='Server Response Times', label='KDE pandas')\n",
    "sns.kdeplot(server_data[0], bw=20, color='black', lw=1.0, label='KDE seaborn')\n",
    "\n",
    "#server_data[0].hist(bins=100, normed=True)\n",
    "#server_data[0].plot(kind='kde')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Experiment with the bandwidth parameter for the seaborn KDE until you discover the parameter that matches the KDE rendered by matplotlib directly on the data frame object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seaborn provides a very nice short cut to the above with a single command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(server_data[0], bins=60, kde_kws={\"bw\": 20});\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Cumulative Distribution Function (CDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The advantage of histograms and KDEs is that they have an **intuitive appeal**. They clearly **convey how probable it is to observe a data point with a given value**. \n",
    "\n",
    "From the above example, it is immediately clear that values around 250\n",
    "milliseconds are very likely to occur, whereas values greater than 2,000 milliseconds are rare.\n",
    "\n",
    "The question is, **how rare, exactly?**\n",
    "\n",
    "That is much harder to answer by looking at a histogram in the above example. Aside from wanting to answer how much weight is in the tail, **a very pertinent question might also seek to know exactly what fraction of requests completes in the typical band between 150 and 350 milliseconds.** \n",
    "\n",
    "In order to answer this more easily, we need to **sum up the contributions from all bins in that region**.\n",
    "\n",
    "The cumulative distribution function (CDF) does just that. The **CDF at point x tells us what\n",
    "fraction of observations occurred “to the left” of x**. In other words, the CDF is the fraction of\n",
    "all points $x_i$ with $x_i ≤ x$.\n",
    "\n",
    "As an exercise we can plot a CDF programatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.kdeplot(server_data[0], cumulative=True, label='CDF')\n",
    "server_data.hist(bins=60, density=True, alpha=0.2, color='grey', sharex=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above figure combines both the histogram and the KDE, as well as our CDF.\n",
    "\n",
    "Several interesting observations can be read directly from the plot of the CDF. \n",
    "\n",
    "In the tail of the distribution at *t* = 1,500, the CDF is still smaller than 0.85; this means that 15 percent of all requests take longer than 1,500 milliseconds. \n",
    "\n",
    "In contrast, less than a third of all requests are completed in the *typical* range of 150–500 milliseconds. \n",
    "\n",
    "How do we arrive at this? \n",
    "\n",
    "The CDF for t = 150 is about 0.05 and is close to 0.40 for t = 500. In other words, about 40 percent of all requests are completed in less than 500 milliseconds; of these, 5 percent are completed in less than 150 milliseconds. Hence about 35 percent of all requests have response times of between 150 and 500 milliseconds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is worth pausing to contemplate these findings, because **they demonstrate how misleading a histogram (or KDE) can be despite (or because of) their intuitive appeal!**\n",
    "\n",
    "Judging from the histogram or KDE alone, it seems quite reasonable to assume that “most” of the events occur within the major peak near t = 300 and that the tail for t > 1,500 contributes relatively little. \n",
    "\n",
    "Yet the CDF tells us clearly that this is not so. (The problem is that** the eye is much better at judging distances than areas, and we are therefore misled by the large values of the histogram near its peak and fail to see that nevertheless the area beneath the peak is not that large compared to the total area under the curve**.)\n",
    "\n",
    "CDFs are probably the least well-known and most under appreciated tool in basic graphical analysis. They have less immediate intuitive appeal than histograms or KDEs, but they allow us to make the kind of quantitative statement that is very often required but is difficult (if not impossible) to obtain from a histogram.\n",
    "\n",
    "CDFs have a number of important properties that follow directly from how they are calculated.\n",
    "\n",
    "1. Because the value of the CDF at position x is the fraction of points to the left of x, a CDF is always monotonically increasing with x.\n",
    "2. CDFs are less wiggly than a histogram (or KDE) but contain the same information in a representation that is inherently less noisy.\n",
    "3. Because CDFs do not involve any binning, they do not lose information and are therefore a more faithful representation of the data than a histogram.\n",
    "4. All CDFs approach 0 as x goes to negative infinity. CDFs are usually normalized so that they approach 1 (or 100 percent) as x goes to positive infinity.\n",
    "5. A CDF is unique for a given data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cumulative distribution functions have several uses:\n",
    "\n",
    "1. they enable us to answer questions such as those posed earlier in this section: what fraction of points falls between any two values? The answer can simply be read off from the graph.\n",
    "2. they help us understand how imbalanced a distribution is—in other words, what fraction of the overall weight is carried by the tails.\n",
    "3. CDFs also prove useful when we want to compare two distributions. It is notoriously difficult to compare two bell-shaped curves in a histogram against each other. Comparing the corresponding CDFs is usually much more conclusive.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Only When Appropriate: Summary Statistics and BoxPlots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far not much has been said on such simple topics as mean and median, standard deviation, and percentiles. This has been quite intentional. \n",
    "\n",
    "These summary statistics apply only under certain assumptions and are misleading if those assumptions are not fulfilled. \n",
    "\n",
    "Mean, median, and related summary statistics apply only to distributions that have a\n",
    "single, central peak — that is, to unimodal distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a distribution has only a **single peak**, then it makes sense to ask about the properties of\n",
    "that peak: where is it located, and what is its width? \n",
    "\n",
    "We may also want to know whether the distribution is symmetric and whether any outliers are present.\n",
    "\n",
    "**Mean and standard deviation are easy to calculate**, and have certain nice mathematical\n",
    "properties — **provided the data is symmetric and does not contain extreme outliers**.\n",
    "\n",
    "Unfortunately, **many data sets violate at least one of these assumptions**. \n",
    "\n",
    "By calling *describe()* on a data frame with numerical values, we can very easily extract most of the simple summary statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_data.describe()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Box and Whisker Plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is an interesting **graphical way to represent the summary statistics**, together with\n",
    "information about potential **outliers**. This plot is known as a box-and-whisker plot, or box plot for short.\n",
    "\n",
    "Figure below illustrates the components of a box plot on the server data. \n",
    "\n",
    "A box plot consists of:\n",
    "\n",
    "1. A marker or symbol for the median as an indicator of the location of the distribution\n",
    "2. A box, spanning the inter-quartile range, as a measure of the width of the distribution\n",
    "3. A set of whiskers, extending from the central box to the upper and lower adjacent values, as an indicator of the tails of the distribution (where “adjacent value” is defined below)\n",
    "4. Individual symbols for all values outside the range of adjacent values, as a representation for outliers\n",
    "\n",
    "While the inter-quartile range is a measure for the width of the central\n",
    "“bulk” of the distribution, the adjacent values are one possible way to express how far its\n",
    "tails reach. The upper adjacent value is the largest value in the data set that is less than\n",
    "twice the inter-quartile range greater than the median. In other words: extend the\n",
    "whisker upward from the median to twice the length of the central box. Now trim the\n",
    "whisker down to the largest value that actually occurs in the data set; this value is the\n",
    "upper adjacent value. (A similar construction holds for the lower adjacent value.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = server_data.boxplot()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.boxplot(server_data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log(wine_data.loc[:,'Alcohol':]).plot(kind='box')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_data[['Class','Alcohol']].boxplot(by='Class')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Generate a box-and-whisker plot on the Wine dataset for the Malic acid feature, and generating the distributions by class label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Plot a vertical set of boxplot with superimposed points as above on the tips dataset, showing the distribution of tip percentages by sex:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add a third dimension again using hue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rcParams['figure.figsize'] = 15, 10\n",
    "# Draw a nested boxplot to show bills by day and sex\n",
    "sns.boxplot(x=\"day\", y=\"total_bill\", hue=\"sex\", data=tips, palette=\"PRGn\")\n",
    "sns.despine(offset=10, trim=True)\n",
    "\n",
    "plt.title('Distribution of Total Bill values for each Day by Sex \\n for the Tips Dataset', alpha=0.75, fontsize=30)\n",
    "plt.ylabel('Total Bill', alpha=0.75)\n",
    "plt.xlabel('Day', alpha=0.75)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Draw a nested violinplot and split the violins for easier comparison\n",
    "sns.set(style=\"whitegrid\", palette=\"pastel\", color_codes=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A violin plot plays a similar role as a box and whisker plot. It shows the distribution of quantitative data across several levels of one (or more) categorical variables such that those distributions can be compared. Unlike a box plot, in which all of the plot components correspond to actual datapoints, the violin plot features a kernel density estimation of the underlying distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "rcParams['font.size'] = 20\n",
    "sns.violinplot(ax=ax, x=\"day\", y=\"total_bill\", hue=\"sex\", data=tips, split=True,\n",
    "               inner=\"quart\", palette={\"Male\": \"b\", \"Female\": \"y\"})\n",
    "sns.despine(left=True)\n",
    "plt.title('Distribution of Total Bill values for each Day by Sex \\n for the Tips Dataset', alpha=0.75, fontsize=30)\n",
    "plt.ylabel('Total Bill', alpha=0.75)\n",
    "plt.xlabel('Day', alpha=0.75)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bar Graphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matplotlib.style.use('ggplot')\n",
    "rcParams['figure.figsize'] = 15, 10\n",
    "rcParams['font.size'] = 20\n",
    "rcParams['axes.facecolor'] = 'white'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_data.loc[:,'Alcohol':].mean().plot(kind='bar')\n",
    "plt.xticks(rotation=0, fontsize=20)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FactorPlots draw a categorical plot onto a FacetGrid.\n",
    "\n",
    "The default plot that is shown is a point plot, but other seaborn categorical plots can be chosen with the kind parameter, including box plots, violin plots, bar plots, or strip plots.\n",
    "\n",
    "It is important to choose how variables get mapped to the plot structure such that the most important comparisons are easiest to make. As a general rule, it is easier to compare positions that are closer together, so the hue variable should be used for the most important comparisons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Draw a nested barplot to show survival for class and sex\n",
    "ax = sns.catplot(x=\"day\", y=\"percentage\", hue=\"smoker\", data=tips,\n",
    "                   height=10, kind=\"bar\", palette=\"muted\")\n",
    "ax.despine(left=True)\n",
    "ax.set_ylabels(\"Tip Percentage of Total Bill\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = sns.catplot(x=\"day\", y=\"percentage\", hue=\"smoker\", data=tips,\n",
    "                   height=10, kind=\"strip\", palette=\"muted\")\n",
    "ax.despine(left=True)\n",
    "ax.set_ylabels(\"Tip Percentage of Total Bill\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scatter Plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scatter plots show us relationships between variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tips.columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(tips.total_bill, tips.tip)\n",
    "plt.ylabel('Total bill')\n",
    "plt.xlabel('Tip')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot a third variable as a density of the individual dots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tips.plot(kind='scatter', x='total_bill', y='tip', c='size', s=100);\n",
    "plt.ylabel('Tip')\n",
    "plt.xlabel('Total bill')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, we can also plot a third variable as the volume of the dots:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tips.plot(kind='scatter', x='total_bill', y='tip', s=tips['percentage']*20, alpha=.75);\n",
    "plt.ylabel('Tip')\n",
    "plt.xlabel('Total bill')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.plotting import figure, output_notebook, show\n",
    "output_notebook()\n",
    "p2 = figure(title=\"Total bill, Tip Amount and Tip Percentage\")\n",
    "p2.scatter(tips.total_bill, tips.tip, color='#A6CEE3', legend='Tip Percentage',size=tips.percentage,alpha=.75)\n",
    "\n",
    "p2.grid.grid_line_alpha=0.3\n",
    "p2.xaxis.axis_label = 'Total bill'\n",
    "p2.yaxis.axis_label = 'Tip'\n",
    "show(p2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Plot a scatter plot on the Wine dataset, showing Alcohol and Ash on the x and y axes, and Alcalinity of ash reflected as the size of the dots.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Render the previous plot in Bokeh as an interactive plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = sns.FacetGrid(wine_data,  hue=\"Class\", height=10)\n",
    "g.map(plt.scatter, \"Alcohol\", \"Ash\", s=200, edgecolor=\"w\").add_legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Referring back to the FacetGrids shown earlier using histograms, by using hue we can represent the third variable and generate scatter plots instead: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = sns.FacetGrid(tips, col=\"smoker\",  hue=\"sex\", height=6)\n",
    "g.map(plt.scatter, \"total_bill\", \"percentage\", s=100, edgecolor=\"w\").add_legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Plot a FacetGrid figure depicting the relationship between the total bill and the tip size, split into two graphs based on sex, a scatter plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting Multivariate data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often we would like to see the if relationships exist between different features within a dataset. One such plot that can be  easily generated to display correlations between all features in a pair-wise manner is a scatter matrix plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = pd.plotting.scatter_matrix(wine_data[wine_data.columns[1:6]], alpha=1, figsize=(20, 15), diagonal='hist')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pd.tools.plotting.scatter_matrix(wine_data[wine_data.index[1:6]], alpha=1, figsize=(20, 15), diagonal='hist')\n",
    "\n",
    "g = sns.PairGrid(wine_data)\n",
    "g.map_diag(sns.kdeplot)\n",
    "g.map_offdiag(sns.kdeplot, cmap=\"Blues_d\", n_levels=6);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple plot showing the strength of relationship between two features can be rendered below using seaborn's jointplot. This results in a combination of a scatter plot between two variables, a regression line (with a confidence interval) as well as the Pearson's correlation coefficient that indicates the strength of the linear relationship, together with the histograms and the KDE's of each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats as stats\n",
    "\n",
    "j = sns.jointplot(\"Ash\", \"Alcalinity of ash\", wine_data, kind='reg', height=9);\n",
    "j.annotate(stats.pearsonr)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The symbol for Pearson's correlation is \"ρ\" when it is measured in the population and \"r\" when it is measured in a sample. \n",
    "\n",
    "Pearson's correlation coefficient can range from -1 to 1. An r of -1 indicates a perfect negative linear relationship between variables, an r of 0 indicates no linear relationship between variables, and an r of 1 indicates a perfect positive linear relationship between variables. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Select a pair of features from the above example and generate a jointplot on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "---\n",
    "\n",
    "---\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reference material"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3d Plotting "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mpl_toolkits.mplot3d.axes3d import Axes3D\n",
    "import matplotlib.pyplot as plt\n",
    "from numpy import *\n",
    "import matplotlib as mpl\n",
    "\n",
    "def flux_qubit_potential(phi_m, phi_p):\n",
    "    return 2 + alpha - 2 * cos(phi_p)*cos(phi_m) - alpha * cos(phi_ext - 2*phi_p)\n",
    "\n",
    "fig = plt.figure(figsize=(14,6))\n",
    "\n",
    "alpha = 0.7\n",
    "phi_ext = 2 * pi * 0.5\n",
    "\n",
    "\n",
    "phi_m = linspace(0, 2*pi, 100)\n",
    "phi_p = linspace(0, 2*pi, 100)\n",
    "X,Y = meshgrid(phi_p, phi_m)\n",
    "Z = flux_qubit_potential(X, Y).T\n",
    "\n",
    "# `ax` is a 3D-aware axis instance because of the projection='3d' keyword argument to add_subplot\n",
    "ax = fig.add_subplot(1, 2, 1, projection='3d')\n",
    "\n",
    "p = ax.plot_surface(X, Y, Z, rstride=4, cstride=4, linewidth=0)\n",
    "\n",
    "# surface_plot with color grading and color bar\n",
    "ax = fig.add_subplot(1, 2, 2, projection='3d')\n",
    "p = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=mpl.cm.coolwarm, linewidth=0, antialiased=False)\n",
    "cb = fig.colorbar(p, shrink=0.5)\n",
    "\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "matplotlib is a relatively low-level plotting package, relative to others. It makes very few assumptions about what constitutes good layout (by design), but has a lot of flexibility to allow the user to completely customize the look of the output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting Plot Parameters and annotating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = arange(0.,10.,0.1) # generate a range of values as an array, using begin, end, step as input \n",
    "y = sin(x)\n",
    "ll = plt.plot(x,y)\n",
    "xl = plt.xlabel('horizontal axis')\n",
    "yl = plt.ylabel('vertical axis')\n",
    "ttl = plt.title('sine function')\n",
    "ax = plt.axis([-2, 12, -1.5, 1.5])\n",
    "grd = plt.grid(True)\n",
    "txt = plt.text(0,1.3,'here is some text')\n",
    "ann = plt.annotate('a point on curve',xy=(4.7,-1),xytext=(3,-1.3),arrowprops=dict(arrowstyle='->'))\n",
    "line = plt.plot(x,y)[0]\n",
    "\n",
    "#to set markers on the points\n",
    "#line.set_marker('o')\n",
    "\n",
    "#save file\n",
    "plt.savefig('sine_function.png',dpi=150)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = arange(0,3,.02)\n",
    "b = arange(0,3,.02)\n",
    "c = exp(a)\n",
    "d = c[::-1]\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(a,c,'k--',a,d,'k:',a,c+d,'k')\n",
    "leg = ax.legend(('Model length', 'Data length', 'Total message length'),\n",
    "           'upper center', shadow=True)\n",
    "ax.set_ylim([-1,20])\n",
    "ax.grid(True)\n",
    "ax.set_xlabel('Model complexity --->')\n",
    "ax.set_ylabel('Message length --->')\n",
    "ax.set_title('Minimum Message Length')\n",
    "\n",
    "ax.set_yticklabels([])\n",
    "ax.set_xticklabels([])\n",
    "\n",
    "# set some legend properties.  All the code below is optional.  The\n",
    "# defaults are usually sensible but if you need more control, this\n",
    "# shows you how\n",
    "\n",
    "# the matplotlib.patches.Rectangle instance surrounding the legend\n",
    "frame  = leg.get_frame()\n",
    "frame.set_facecolor('0.80')    # set the frame face color to light gray\n",
    "\n",
    "# matplotlib.text.Text instances\n",
    "for t in leg.get_texts():\n",
    "    t.set_fontsize('small')    # the legend text fontsize\n",
    "\n",
    "# matplotlib.lines.Line2D instances\n",
    "for l in leg.get_lines():\n",
    "    l.set_linewidth(1.5)  # the legend line width\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "require(['base/js/utils'],\n",
    "function(utils) {\n",
    "   utils.load_extensions('calico-spell-check', 'calico-document-tools', 'calico-cell-tools');\n",
    "});"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
